stanfordfandomcom-20200214-history
CS349W Project3 jGestures
CS349W project 3 writeup Introduction (D) We have developed jGestures (code), a jQuery plugin that enables mouse gestures in web applications. Mouse gestures have been available in traditional single-user applications refs and even some browsers opera refs, but only for browser commands (e.g. next/previous page, home, etc). jGestures lets web developers integrate mouse gestures into their web applications easily and efficiently. Its main features are: * Runs completely on the client side, using JavaScript. * Highly configurable: Developers specify their own gestures, can customize the way gestures are drawn, and can tweak the recognition algorithm. * Fast, small code footprint (3.6KB minified). * Easy to integrate as a jQuery plug-in. 'Basic operation:' jGestures handles gesture capture, recognition, and draws ..., the web developer specifies a set of gestures and and a gesture handler for each. TODO: Describe operation Explain how the plug-in works in more detail (how user draws gesture, colors of lines, upon finishing gesture recognized, feedback by re-coloring gesture, fire event-handler). Interface (D) Integrating jGestures with your web application is very simple: # Include the jQuery and jGestures JavaScript files. #: #: #: # Optionally, modify the default configuration (see all the parameters in the reference) #: #: $(this).gesturesConfig().strokeWidth = 5; # Add your own gestures (see specifying gestures) #: #: var gHandler = function (gesture,points) { #: alert("Detected: " + gst.name); #: }; #: #: $(this).addGesture( #: [0,0, 0,1, 1,1, 1,0, 0,0], #: false, false, false, "Box", gHandler #: ); #: #: $(this).addGesture( #: [0,0, 1,0], #: true, true, true, "L-R stroke", gHandler #: ); # Initialize the jGestures plug-in #: $(this).initGestures(); Specifying gestures We now discuss the different possibilities when adding a gesture. The addGesture signature takes the following parameters: $(this).addGesture(points, directionSensitivity, proportionSensitivity, startSensitivity, name, handler); * points: Sequence of points that form the gesture. Can be in any coordinate system * Sensitivity options (see figure): ** directionSensitivity: Whether the direction in which the gesture is drawn matters ** proportionSensitivity: Whether the proportions of the gesture matter ** startSensitivity: Whether the starting point of the gesture matters (useful for closed gestures, e.g. boxes or circles) * name: Name of the gesture * handler: Event handler that will be called when the gesture is detected. The handler takes two arguments: the gesture detected, and the sequence of points captured. jGestures Use Cases To demonstrate some possible applications of mouse gestures and evaluate its ease of use, we have integrated it with two applications: Galleria The jQuery galleria plugin implements a browsable photo library. Mouse gestures can be used to traverse the gallery. Gestures are drawn by pressing the left button and holding it while drawing the gesture. Multiple gestures can be used: * Left-right stroke to go to the next image, right-left to go to the previous one * Up-down stroke to go to the first image, down-up to go to the first one * Up-down + left-right strokes (an "L") to jump to the next 5 images, and up-down + right-left to jump the the previous 5. This integration required minimal effort (10 lines of code). The Galleria demo described can be experienced here. Google Maps We used the Google Maps API to create a map in which the zoom level can be controlled using mouse gestures. The recognized gesture, a box around a specific region of the map, causes the map to zoom and focus region. To draw the box, single click on the map, trace a rectangular shape around your desired region, then single click again to end the gesture. The map will zoom in to the outlined area. Integrating gestures with Google Maps required somewhat more effort because of conflicts with the event handlers provided by the Google Maps API. The Google Maps demo can be experienced here. Implementation (D) We now discuss how jGestures captures and recognizes gestures. # Capture: By default, when initialized jGestures installs handlers for the mousedown, mousemove and mouseup events. If the developer wishes to implement custom handler or these default handlers disrupt normal operation, the methods gestureStart(x,y), gestureUpdate(x,y) and gestureEnd(x,y) can be used instead. In general, once a gesture is recorded it will be difficult to compare it to other gestures, as it contains an arbitrary number of points. To solve this, every gesture is normalized. # Normalization is a two-step process: first, the gesture is scaled and translated so that every point is in the range 0.400x0,400. Then, the gesture is re-sampled to contain a fixed number of points at # Comparison: The normalized gesture is compared with all the gestures specified (which have also been normalized). We compute the following distance metric: #:To allow for gestures that are not sensitive to the starting point, we first find the closest point to a(0) in b, and rotate b so that we consider that point as our origin, b(0). For gestures that are insensitive to the direction, we reverse b if dist(a(1), b(1)) < dist(a(1), b(n-1)). - Mouse handlers - Normalization - Comparison - Drawing Discussion and Evaluation The jGestures plugin is a natural and interactive way to control certain web applications, as shown in the two demos above. The recognition algorithm works quite well, and we have seen good performance in terms of the amount of time required per recognition (around 2.5 ms if one does not including the drawing). The entire plugin is small (about 3.6 KB minified), and its representation as a jQuery extension allows one to utilize the entire library and ease-of-use provided by jQuery. One point of discussion is whether or not mouse gestures are a useful way to interact within a web application. For example, some might say that in the Galleria application, it would be just as easy to have buttons/links to traverse the pictures as opposed to gestures. However, our particular implementation allows detection of closed polygons such as squares. This gives a developer the flexibility to add new interactions such as the zoom feature demonstrated in the Google Maps example. In addition, we feel that with the rise of embedded machines such as the iPhone, there is ample opportunity for an error-tolerant interaction such as mouse gestures. In creating the mouse gestures implementation, we experienced a few issues. The drawing feature was simple in theory, but there is currently lack of browser support for a high-performance drawing canvas that is overlaid on top of other elements. Thus we had to turn to a more resource-intensive solution, which basically colored tiny div elements and used these to compose lines. We also had issues with integrating our plugin with the Google Maps API. The Google Maps API uses its own set of mouse event handlers, and these conflicted with the handlers that we encapsulated in the jGestures plugin. We solved the problem by using the Google Maps handlers for that particular application. source code link? API and Configuration Options API One can be up and running with a simple "left to right" gesture by adding the following code: $(this).addGesture([ 0,0 , 1,0 ], true, true, true, "l_to_r", handler_func); $(this).initGestures(); Function Arguments Description addGesture coordinates, direction-sensitive, start-sensitive, proportion-sensitive, name, handler Describe a gesture initGestures None Registers the event handlers to allow for gesture recognition g_mouseDown None Event handler. Begins recording coordinates and prepares for drawing g_mouseMove None Event handler. Records coordinate and continues drawing g_mouseUp None Event handler. Runs matching algorithm on points, determines result, and clears drawing. Configuration Configuration of the jGestures plugin is available via modification of the config object. This object can be accessed by calling gesturesConfig(). For example, changing the number of points to 30 would simply require the following code: $(this).gesturesConfig().numPoints = 30; Config Option Default Value Description numPoints 40 Number of coordinates used in the internal representation of each gesture maxThreshold 5000*numPoints Allowable difference in sum of square deltas - any greater error results in no match xLen, yLen 400, 400 Dimensions of canvas surface to which all gestures are normalized registerDefaultHandlers true Determines whether to register default jGestures handlers mouseButton 0 Mouse button to use for gestures (firefox only): 0-left, 1-middle, 2-right drawGesture true Specifies whether to visibly draw gestures with ink strokeColor yellow Valid if drawGesture is true. Determines the color of the ink used strokeWidth 3 Integer describing the width of ink used to draw gestures detectedColor green Drawing ink changes to this color if a gesture is matched notDetectedColor red Drawing ink changes to this color if no gesture is matched afterDetectionDisplay 200 (ms) Number of ms to show detected ink color after finishing gesture